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ABSTRACT 



This study was commissioned to investigate issues related to 
the inclusion of the American College Testing Program (ACT) assessment within 
the Colorado State Assessment Program (CSAP) . Analyses were carried out using 
assessment materials, assessment results, and various data from the ACT and 
the Colorado State Department of Education. One focus of the investigation 
was the match between the Colorado Model Content Standards (CMCS) and the ACT 
Assessment. Data indicate that the ACT does not match well with the CMCS, and 
the match is especially poor in science. This finding casts doubts on the 
advisability of requiring all Colorado 11th graders to take the ACT. The poor 
match could potentially inject incoherence into the CMCS -based accountability 
program as a whole, and it raises questions about how ACT results might be 
combined with CSAP performance in the calculation of Student Accountability 
Reports for high schools. (SLD) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



TM034215 ED 466 476 



* 







IMPLICATIONS FOR USE OF THE ACT WITHIN THE 
COLORADO STUDENT ASSESSMENT PROGRAM 




Damian Betebenner and Ken Howe 
School of Education 
University of Colorado 
Boulder, CO 



December 15, 2001 




PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 

j) lOdCHon 




TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



BEST COPY AVAILABLE 



^UaDEPARTMENT OF EDUCATION 

Office of Educational Research and Improvement 

educational resources information 

. CENTER (ERIC) 

SA This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



2 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 

~ ,y 



t X 



Executive Summary 

The study described here was commissioned by the Colorado Department of Education 
(CDE). Its purpose, as elaborated in RFP-MR-ASMT-0001, is to investigate issues related 
to the inclusion of the ACT assessment within the Colorado State Assessment Program. 
Findings are reported in six categories: 

• A statistical summary of the ACT assessment during the 1998-99, 1999-00, and 2000-01 
school years for 11th graders in the state of Colorado. 

• An analysis of how well the ACT assessment measures content knowledge in reading, 
writing, mathematics, and science as defined for grades 9-12 in the Colorado Model 
Content Standards (CMCS). 

• A comparison of the ACT to the SAT I and the SAT II with respect to the CMCS. 

• An examination of the costs associated with the development of 11th grade Colorado 
State Assessment Program (CSAP) exams in reading, writing, mathematics, and sci- 
ence. 

• A comparison of diagnostic reporting between the ACT and the CSAP exams. 

These analyses were carried out using assessment materials, assessment results, and various 
data defined in the body of the report and supplied by ACT Inc. and CDE. 

Summary results for the ACT Results for 11th graders taking the ACT during the 
1998-99, 1999-00, and 2000-01 academic years show that those juniors taking the exam 
during the first two academic years are a select group, with their means close to 1 
point above the state graduating class mean of 21.5. With the inclusion of all juniors 
in 2000-01, the mean drops nearly two points below the state graduating class average 
to 19.6. When data were disaggregated by gender and ethnicity, similar decreases in 
mean ACT composite scores were noted with the inclusion of all juniors in 2000-01. 
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The ACT assessment as a measure of the CMCS Using assessment frameworks for 
the 10th grade CSAP exams in reading, writing, and mathematics and the CMCS 
in science, all items on two retired ACT assessments were mapped to the CMCS stan- 
dard that they best fit. Using the 10th grade CSAP exams as exemplars of tests 
meeting the CMCS, we found that the ACT tests fell short of being tests of whether 
students understood the CMCS. The worst performing test in this regard was the ACT 
science reasoning test, which fails to assess the advanced science content present in the 
CMCS. 

The ACT mathematics, reading, and writing tests better matched the content dictated 
by the CMCS than the science reasoning test but fell short in a number of areas 
compared to the exemplar 10th grade CSAP examinations in mathematics, reading, 
and writing. Specifically, the ACT and CSAP tests showed different emphases on the 
CMCS standards comprising each of the subject areas. There were also differences 
among ACT and CSAP test items that mapped to the same standard. The CSAP 
items were often more demanding than their ACT counterparts, requiring students to 
go beyond just finding the right answer to also explain their answers in written form. 

Alternates to the ACT assessment There are currently two alternates to the ACT as- 
sessment in widespread use: the SAT I and SAT II. Unlike the ACT assessment, the 
SAT I is not curriculum based and would likely be a worse match than the ACT to 
the CMCS. Unlike the SAT I, the SAT II achievement tests are curriculum based. It 
is unclear whether they would map to the CMCS better than the ACT. Unlike the 
ACT and SAT I, which serve as the admissions tests to the majority of colleges and 
universities in the United States, the SAT II tests are usually only required at elite 
schools and in conjunction with either the ACT or SAT I. 

Costs associated with the development of 11th grade CSAP exams The estimated 
costs associated with the development of the 11th grade CSAP exams in reading, writ- 
ing, science, and mathematics for 45,000 students provided by CTB /McGraw-Hill are 
$4,600,000 in the first year and $3,900,000 in subsequent years. By comparison, the 
price charged by ACT Inc. for administration of the ACT in 2001 is $23 per student 
resulting in total costs of $1,035,000. 

Diagnostic reporting associated with the ACT assessment and CSAP exams The 
diagnostic reports associated with the ACT and CSAP exams differ in two important 
respects: 

1. Diagnostic reports for the CSAP exam provide performance level results to stu- 
dents, school, and districts that are clearly tied to the CMCS. The ACT diagnostic 
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reports, both for students and schools, provide composite score data, scores on 
each of the four tests, and scores on subareas tested within each test. Normative 
data is also supplied to help interpret the scores. 

2. The school, district, and state reports for the ACT are given for the graduating 
class whereas the CSAP reports provide information on students taking the test 
during the previous academic year. 

Conclusions The ACT does not match well with the CMCS, and the match is especially 
poor in science. This finding casts serious doubt on the foremost rationale for requiring 
all Colorado 11th graders to take the ACT, namely, that the ACT provides a good 
measure of how well students have mastered the CMCS. It should be noted that several 
other rationales are not impugned by the finding of a poor match between the ACT 
and the CMCS. These rationales are (1) that requiring all Colorado 11th graders to 
take the ACT might encourage more students to pursue a college education and (2) 
that taking the ACT in their junior years might help students identify deficiencies that 
they might remedy in their senior years. 

That the ACT is poorly matched with the CMCS has two further implications-. First, it 
potentially injects incoherence into the CMCS-based accountability system as a whole. 
Because the CSAPs and the ACT measure different things, they are potentially at cross 
purposes. That is, teaching that concentrates on increasing performance on the CSAP 
exams could come at the cost of lowering performance on the ACT, and vice versa. 
Second, and related to the first problem, the poor match raises serious question about 
how ACT results might be combined with CSAP performance in the calculation of 
Student Accountability Reports for high schools. Because the reports are finely tuned 
to provide information related to performance on the CMCS, it is unclear how to amend 
the reports’ format to include ACT performance without reducing the reports’ focus 
on performance with respect to the CMCS. 

One important finding pertaining to an issue that was not included in the original aim 
of the study emerged through the process of data analysis. In particular, the 2002 
ACT graduating class summary will indicate a decrease of approximately 2.5 points 
compared to 2001. This is an outcome of the manner in which ACT Inc. reports results 
coupled with all Colorado juniors being required to take the exam in 2001 (placing them 
in the 2002 graduating class). This decrease should be interpreted with some caution 
since it is largely the result of a different population of students in Colorado taking 
the ACT. Comparisons to previous Colorado ACT graduating class results or to other 
states’ ACT results based upon self-selecting ACT examinees are tenuous. 
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Introduction 



This report was commissioned by the Colorado Department of Education (CDE) in accor- 
dance with RFP-MR-ASMT-0001 (hereafter referred to as RFP) to investigate issues related 
to Senate Bill 00-186, which mandates that all 11th grade students in Colorado take the ACT. 
This report includes analyses outlined in the proposal submitted by the authors of this study 
in response to the RFP. 

The ACT Assessment (hereafter referred to as the ACT) is a curriculum-based exam 
developed and administered by ACT Inc. It consists of four tests, English, mathematics, 
reading, and science reasoning. The test’s primary use is as an entrance exam for colleges 
and universities throughout the United States. With the passage of Senate Bill 00-186, all 
11th grade students in the state of Colorado, except in a few special circumstances, are 
required to take the ACT in the spring of their junior year. Prior to 2001, the exam was 
largely voluntary with a few schools opting to pay for the ACT to be administered to all 
their students. 

CDE currently administers the Colorado State Assessment Program (CSAP) with exams 
constructed in accordance with the Colorado Model Content Standards (CMCS). As of the 
2001-2002 school year, exams are given to various grades in various content areas. The 
schedule associated with the CSAP is presented in Table 1. 



Grade 3 


Reading and writing 


Grade 4 


Reading and writing 


Grade 5 


Reading, writing, and mathematics 


Grade 6 


Reading, writing, and mathematics 


Grade 7 


Reading, writing, and mathematics 


Grade 8 


Reading, writing, mathematics, and science 


Grade 9 


Reading, writing, and mathematics 


Grade 10 


Reading, writing, and mathematics 


Grade 11 


ACT 



Table 1: CSAP exam schedule for grade 3 to 11 with associated subjects 



The CSAP exams given in grades 3 through 10 are developed by CTB/McGraw-Hill and 
are specifically designed to assess student mastery of designated subject matter as defined 
within the CMCS. The ACT , on the other hand, is not constructed to the CMCS but instead 
uses the Educational Planning and Assessment System (EPAS) standards developed by ACT 
Inc. as its template. This study was commissioned to investigate issues related to the use 



of the ACT assessment as a measure of the CMCS in reading, writing, mathematics, and 
science. 

As outlined in the proposal offered in response to the RFP, this report presents results 
of an investigation of issues related to the use of the ACT assessment in the following six 
categories: 

• A statistical summary of ACT assessment results during the 1998-99, 1999-00, and 
2000-01 school years for 11th graders in the state of Colorado. 

• An analysis of how well the ACT assessment measures content knowledge in reading, 
writing, mathematics, and science as defined for grades 9-12 in the CMCS. 

• A comparison of the ACT to the SAT I and the SAT II with respect to the CMCS. 

• An examination of the costs associated with the development of 11th grade CSAP 
exams in reading, writing, mathematics, and science. 

• A comparison of diagnostic reporting between the ACT and the CSAP ex ams . 

Summary of ACT data for years 1998-99, 1999-00, and 
2000-01 



This section examines the results on the ACT examination for 11th graders during the 1998- 
99, 1999-00, and 2000-01 academic years. Because Senate Bill 00-186 requires all 11th grade 
students to take the ACT beginning in spring 2001, summary results of the 2000-01 ACT 
data on juniors is not comparable to the previous two academic years, when juniors taking 
the test did so voluntarily. 1 Moreover, the 2000-01 data includes only those juniors taking 
the test during the mandatory dates. The, summary results reported include number of 
11th grade students taking the ACT, mean, median, and standard deviation of the ACT 
composite scores. The data used for these analyses included only valid scores as reported 
to us by ACT Inc. In 2000-01, juniors absent and without a legitimate exemption from 
the ACT received scores of zero. Differences in results reported here and those available 
on the CDE website are largely attributable to this difference in accounting for absentees. 
In addition to statewide results for juniors in each of the three academic years, results are 
disaggregated by gender, ethnicity, school, and district. Data used for the analyses described 
in this section were provided by ACT Inc. and CDE. 

‘Some schools during the 1998-99 and 1999-00 academic years provided for all of their students to take 
the ACT at school expense during a schoolwide administration. Results for these students are included in 
the 1998-99 and 1999-00 ACT data and summary analyses. 
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Year 


ACT Composite score summary 


1998-99 


mean 


22.50 


median 


22.00 


std. dev. 


4.79 


number tested 


8,381 


1999-00 


mean 


22.42 


median 


22.00 


std. dev. 


4.78 


number tested 


9,912 


2000-01 


mean 


19.63 


median 


19.00 


std. dev. 


5.09 


number tested 


41,358 



Table 2: Statewide summary statistics for 11th grade ACT takers by academic year 



Statewide results 

Results for 11th grade students are presented in Table 2. The decrease in the mean in 2000- 
01 is clearly indicative of a different set of juniors taking the test than in the previous two 
years. Those students taking the ACT as juniors in 1998-99 and 1999-00 are likely high 
achievers who take the test early. The 11th graders taking the ACT in spring 2001 includes 
a similar group of high achievers together with students who would have taken the ACT as 
seniors as well as those students who likely would have never taken the ACT. The increase 
in standard deviation to 5.09 in 2000-01 indicates a more diverse group taking the exam. 



Results by gender and ethnicity 

The results from Table 2 disaggregated by gender and ethnicity are presented in Tables 3 
and 4, respectively. For each gender, the results follow the same pattern as in Table 2: When 
all juniors are tested in 2000-01, the mean and median scores decrease. The same holds true 
with respect to ethnicity. Interestingly, whereas in 1998-99 and 1999-00 males outperformed 
females, in 2000-01, females scored higher than males. 
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Year 


Gender 


ACT Composite score summary 


1998-99 


Female 


mean 


22.38 


median 


22.00 


std. dev. 


4.72 


number tested 


4,600 


Male 


mean 


22.67 


median 


23.00 


std. dev. 


4.87 


number tested 


3,723 


1999-00 


Female 


mean 


22.30 


median 


22.00 


std. dev. 


4.70 


number tested 


5,406 


Male 


mean 


22.57 


median 


22.00 


std. dev. 


4.87 


number tested. 


4,423 


2000-01 


Female 


mean 


19.80 


median 


19.00 


std. dev. 


4.95 


number tested 


20,766 


Male 


mean 


19.47 


median 


19.00 


std. dev. 


5.22 


number tested 


20,408 



Table 3: Statewide summary statistics for 11th grade ACT takers by academic year and 
gender 
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Year 


Ethnicity 


ACT Composite score summary 




African 


mean 


19.06 




American 


number tested 


181 




Native 


mean 


20.66 




American 


number tested 


41 




White 


mean 


22.86 


1998-99 


number tested 


6,463 


Hispanic 


mean 


19.40 




number tested 


565 




Asian 


mean 


23.08 




American 


number tested 


242 




Other 


mean 


21.28 




number tested 


304 




African 


mean 


18.78 




American 


number tested 


193 




Native 


mean 


21.00 




American 


number tested 


56 




White 


mean 


22.88 


1999-00 


number tested 


7,523 


Hispanic 


mean 


19.19 




number tested 


756 




Asian 


mean 


22.11 




American 


number tested 


311 




Other 


mean 


21.02 




number tested 


396 




African 


mean 


16.46 




American 


number tested 


1,174 




Native 


mean 


17.93 




American 


number tested 


412 




White 


mean 


20.78 


2000-01 


number tested 


22,859 


Hisnanic 


mean 


16.49 




JlllU L/Cvlil v 


number tested 


4,415 




Asian 


mean 


19.64 




American 


number tested 


1,207 




Other 


mean 


18.91 






number tested 


2,053 



Table 4: Statewide summary statistics for 11 th grade ACT takers by academic year and 
ethnicity 5 
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Year 


School/District 


ACT Coin 


msite mean 


Minimum 


Maximum 


1998-99 


School (N = 123) 


15.16 


25.58 


District ( N = 58) 


18.45 


25.16 


1999-00 


School (N = 139) 


14.84 


25.79 


District ( N = 67) ' 


18.11 


25.58 


2000-01 


School (N = 271) 


13.72 


26.30 


District ( N = 143) 


14.68 


23.06 



Table 5: Statewide ACT composite score ranges for 11th grade ACT takers by school and 
district 



Results by school and district 

Eleventh grade results aggregated by school and district are presented in Table 5. In line 
with ACT results disseminated by CDE, results presented are for schools/districts with more 
than 15 reported scores for 11th graders. In comparing school level summaries based upon 
data supplied by ACT Inc. in 2000-01 to those available from CDE, significant discrepancies 
exist that could not be reconciled. For example, the school level summary available from 
CDE provides results for the Adult Education/Lincoln Center indicating 76 tested students 
with a mean of 3.61. The lowest individual score in the data provided by ACT Inc. was 
4. Upon consultation with CDE, the discrepancies are the result of absentee test takers 
being assigned a score of 0 by CDE. Of the approximately 40,000 juniors taking the ACT 
in 2001, there were 2,000 students who were absent who were assigned scores of 0. This 
results in state means of these junior ACT takers of 19.6 to 18.5, from ACT Inc. and CDE, 
respectively. 



ACT alignment with the Colorado Model Content Stan- 
dards 

The most important aspect of this study is its investigation of whether the ACT is an 
appropriate assessment to determine the extent to which students, schools, districts, and the 
state overall are meeting the CMCS. The CSAP tests administered in grades 3 through 10 
are specifically designed to test the performance of students against these standards. The 
ACT is not designed according to these exact standards and thus some question exists as 
to whether the ACT can serve as an assessment of 'the CMCS. This section presents our 
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findings with respect to this question. 

ACT Inc. investigated this question in their own study and found that the ACT can 
indeed serve as a measure of how well students are performing with respect to the CMCS 
in all of the four areas that the ACT tests: mathematics, writing (English), reading, and 
science reasoning (American College Testing (ACT), 2001). Because our study comes to 
different conclusions, it bears mentioning how the ACT Inc. study came to those conclusions 
and why we believe that those conclusions are dubious. 

In addressing the question of how well the ACT measures achievement on the CMCS, the 
ACT Inc. study investigated the relationship between the ACT and the CMCS in two steps. 
First, the CMCS are identified which are either “directly” or “indirectly” measured by the 
ACT (ACT, 2001, p. 14). Second, standards on which the ACT assessment is based are iden- 
tified which represent “explicit” or “implied” prerequisite skills defined in the CMCS (ACT, 
2001, p. 14). In our opinion, this method of matching the ACT framework to the CSAP 
framework is liable to an unacceptable level of imprecision and, thus, translation error. First, 
how to determine what is measured “indirectly” or is “implied” is nowhere explained. Sec- 
ond, the ACT items themselves are not examined to determine how well they exemplify 
(or fail to) the CMCS. Our study uses the approach of mapping ACT items individually to 
CMCS standards. The details of how this was done are provided next. 

Method 

To address how well ACT items map to CMCS standards, we began by first specifying 
exactly which standards to attempt to map to. Using the 2001 10th grade CSAP tests in 
reading, writing, and mathematics as our model (since those tests are designed according to 
the CMCS), we acquired from the CDE website the assessment frameworks associated with 
each of the three CSAP exams. These frameworks enumerate what students are expected to 
know and be able to do. In addition, the three CSAP exams are mapped onto this framework 
allowing us to compare how well the reading, writing (English), and mathematics tests of 
the ACT correspond to the three corresponding CSAP exams in terms of emphasis and 
inclusion/exclusion of standards. Since science is only tested in the 8th grade, no assessment 
framework exists for 10th grade science. In order to address how well the ACT science 
reasoning test maps to the science CMCS, we used the science CMCS themselves as the 
assessment framework. 

Using two retired ACT assessments provided by ACT Inc., four subject matter specialists 
were employed to map the items onto the assessment frameworks. The item mappers were 
instructed to associate with each item, if possible, the most specific standard available in the 
associated 10th grade assessment framework. In cases where an item addressed more than one 
standard, the item mapper was instructed to note the standards and rank according to which 
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standard is most clearly addressed. To cross-validate the specialist’s decisions, the science 
and mathematics specialists mapped a random sample of the mathematics and science items 
while the reading and writing specialists mapped a random sample of the writing and reading 
items to the CMOS. When there were disagreements between the mathematics and science 
specialists, the disagreements were easily reconciled after discussion. Disagreements between 
the reading and writing specialists proved more cumbersome. It proved more difficult to 
apply the CMOS in reading and writing to the ACT items and this difficulty manifest 
itself in more disagreements between the two specialists. In all instances the disagreements 
were reconciled and a best standard, if one existed, was determined. In total, 120 ACT 
mathematics items, 150 writing items, 80 reading items, and 80 science reasoning items were 
mapped from the two ACT exams. 

Results 

Upon examination of the item mapping of the science reasoning test, it quickly became 
apparent that the science reasoning test of the ACT could not be mapped onto the high 
school CMCS for science. The science reasoning test incorporates very little in terms of 
advanced science content, whereas the CMCS standards in science dictate that students 
should know very specific and advanced science content in both the physical and life sciences. 
This finding of a mismatch between the ACT test and the standards is at odds with what 
was found in the ACT Inc. study which asserted that, “[t]he EPAS Science Reasoning Tests 
match well with the standards and benchmarks described in the CMCS” (ACT, 2001, p. 5). 

For the other three exams item mapping results were entered into a spreadsheet to fa- 
cilitate the calculation of basic frequencies. Frequencies for the corresponding 10th grade 
CSAP exams were calculated from item maps available on the CDE website. The results 
are summarized for the mathematics, reading, and writing tests of the ACT in Figure l. 2 
It should be noted that those items falling within the no standard category do so because 
they failed to address any standard in the assessment frameworks for the 10th grade CSAP. 
In a number of cases, an item could be mapped onto a standard more elementary than a 
10th grade standard (i.e. , the item addressed a 9th or 8th grade standard). This fact will be 
elaborated on shortly. 

The first two bar charts in Figure 1 demonstrate that the mathematics and reading tests 
of the ACT have different emphases than the corresponding 2001 10th grade CSAP exams. 
Within the mathematics test, for example, the largest discrepancies occurred with standards 

2 The standards associated with the assessment frameworks in each subject area were subdivided beyond 
what is depicted in Figure 1. For example, there is a standard 2.2a in mathematics. Because of the small 
numbers of items in each of sub-standards, aggregates were considered to give a better overall impression of 
the emphasis of each test. 
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ACT and CSAP item content for mathematics 




Colorado Modal Coolant Standard 



ACT and CSAP item content for reading 




Colorado Modal Con tan t Standard 



ACT and CSAP item content for writing 




i * I 

Standard 2 Standard 3 No Standard 

Colorado Model Content Standard 



Figure 1: Bar charts depicting emphasis on the CMOS for the ACT tests and the corre- 
sponding 2001 10th grade CSAP exam 
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3, 4, and 5. Standard 3 emphasizes discrete math in the form of data collection and analysis, 
statistics, and probability; topics which are not taught to a number of high school students 
and which are thus inappropriate for inclusion on the ACT mathematics test. Standard 
5 includes topics involved with measurement in real-world problems making it difficult to 
address with single multiple-choice items like those present in the ACT mathematics test. 
The omission of items addressing standards 3 and 5 is then reflected in the heavy emphasis 
on items which address standard 4, student understanding of geometric concepts, properties, 
and relationships. 

Of the three ACT tests for which frequencies were tabulated, the ACT reading test had 
the most items for which no 10th grade reading standard applied. Fifty-eight percent of 
the 80 reading items examined failed to match a 10th grade standard. A large majority of 
these items did, however, map to a 9th grade standard. Sixty-seven percent of the items 
that failed to map to a 10th grade standard did map to a 9th grade reading standard, most 
often standard 1. 

The English (writing) test of the ACT provided the best match with the CMCS in terms 
of emphasis. This is partly due to the fact that there are only two writing standards in the 
CMCS assessment framework for 10th grade writing. The items of the 10th grade CSAP 
writing exam were almost evenly split between the two standards while 37 percent of the 
ACT items mapped to standard 2 and 52 percent to standard 3. Eleven percent of the ACT 
items matched no standard. 

The preceding analyses delineate the emphases of the ACT and CSAP exams with respect 
to the CMCS but gloss over other important differences between the two exams. Foremost, 
the difference between the ACT tests and the CSAP mathematics, reading, and writing tests 
is a difference in the demands of many of the items. All four of the CSAP exams include open- 
ended response items which require the student to convey their thinking in written form. All 
items on the ACT assessment are multiple choice format. This requirement places greater 
demands on the student to demonstrate their knowledge and results in a more difficult exam; 
one cannot resort to eliminating multiple choice options to answer an open-ended response 
question. Stated succinctly, all test items are not created equal , even if they do address the 
same standard. 

Even when comparing multiple choice items, there was a noticeable difference between 
the difficulty of the items present on the two exams. This became quite apparent when at- 
tempting to map ACT mathematics items to the CMCS. A number of items fit a standard, 
but did so in a less rigorous fashion. For example, math standard 2.2a states that students 
should be able to represent functional relationships using written explanations, tables, equa- 
tions, and graphs and describe the connections among these representations. Examples of 
an ACT and CSAP mathematics item fitting this standard are presented in Figures 2 and 3, 
respectively. 
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How many ordered pairs ( x , y) of real numbers will satisfy the equation 2x - 5y = 6? 

F. 0 

G. 1 

H. 2 

J. 3 

K. Infinitely many 



Figure 2: A sample ACT mathematics item addressing mathematics standard 
2.2a 




Figure 3: A released item on the 10th grade mathematics CSAP bookmarked at 
the proficient level addressing mathematics standard 2.2a 



Though the items address the same standard, the ACT item is not as demanding. The 
CMCS in mathematics dictate that students should be able to solve “real world” problems 
in a number of mathematical contexts. The contrast between the two examples in Figures 2 
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and 3 exemplifies the “real world” emphasis found in many CSAP mathematics items that 
isn’t found in most ACT items. The emphasis on the “real world” application of mathematics 
in the CMOS and the 10th grade CSAP examination leads to the inclusion of many more 
dreaded story problems for students to solve. In short, meeting the CMCS in mathematics 
requires a robust understanding of the subject matter. 



Alternates to the ACT 

Use of college entrance examinations in the United States began at Harvard in the early 
1930s and has grown to the point where nearly every college and university in the country 
requires students to take some test as part of the admissions process. Currently there are 
three exams which predominate: The ACT, the SAT I, and the SAT II. 3 . The utility of 
these exams lies in their ability to predict the first semester GPAs of entering freshmen (i.e., 
predictive validity) which is generally considered a good indicator of college success (Astin, 
1971). Though the tests are used in a similar fashion, the tests are constructed to different 
specifications which merit discussion when considering alternates to the ACT assessment for 
11th graders in Colorado. 

The most important difference between the three tests is that the ACT and SAT II 
are curriculum based and purport to measure student mastery of skills learned in school 
and necessary for success in college studies. By way of contrast, SAT I is an aptitude test 
whose content is not always associated with what one is expected to encounter in high school 
studies. This difference is most apparent on the SAT I verbal exam which includes several 
analogy items (i.e., a is to b as c is to ?). Items of this type do not map onto the CMCS 
making it almost a certainty that were a study to be conducted which looked at how the SAT 
I verbal items mapped to the CMCS, the SAT I would perform much more poorly. Moreover, 
because the SAT I math exam tests math knowledge up to and including geometry, the SAT 
I is highly likely to do worse than the ACT (which includes more advanced mathematics up 
to and including trigonometry) when looked at in the context of the CMCS. Thus, Switching 
from the ACT to one (or both) of the other two exams would likely yields no appreciable 
benefit in terms of getting a better match with the CMCS. 

3 The SAT II achievement tests for assessing knowledge in specific subject areas are required in addition 
to either the ACT or SAT I at some elite universities 
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Costs associated with development of 11th grade CSAP 
exams in reading, writing, mathematics, and science 

Estimates of costs involved with the development of an 11th grade CSAP exam are provided 
by CTB/McGraw Hill, the contractor for the CSAP exams administered in grades 3 to 10. 
The following estimates include costs associated with the development of the four exams, 
scoring of the exams, and the reporting of exam results. The costs are based upon admin- 
istration of the exams to 45,000 students. These costs represent payment to CTB/McGraw 
Hill for services they provide and do not included expenses incurred by other parties to 
administer the exams. 

First year of test $4,600,000 which includes $1,500,000 for development of the four exams 

Subsequent years $3,900,000 which includes $800,000 to produce 25 percent new items 
each year 

By way of contrast, the state of Colorado is charged the national fee by ACT Inc. for any 
students accounted for in state testing. According to data supplied by ACT Inc., the cost 
per student in 2001 was $23. Using the same estimate of 45,000 11th grade students, ACT 
administration costs total $1,035,000. Again, these costs represent payment to ACT Inc. for 
services they provide and do not include expenses incurred by other parties to administer 
the exams. 



Diagnostic reporting between the ACT and CSAP 

Individual diagnostic reports 

ACT Individual students are sent a report indicating their results on the ACT assessment. 
These results include a composite ACT score, four ACT test scores, one each for En- 
glish, mathematics, reading, and science reasoning, and 7 subscores: two subscores for 
English, in usage/mechanics and rhetorical skills; three subscores for mathematics, in 
pre-algebra/ elementary algebra, intermediate algebra/coordinate geometry, and plane 
geometry/trigonometry; and two subscores for reading, in social studies/sciences and 
arts/literature. In addition to the these scores, students are informed of the percent of 
college bound students at or below their given score level. The ACT diagnostic report 
provides no information on performance with respect to the CMCS. 
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CSAP Students receive a student diagnostic report in each of the subject areas in which they 
are tested. The report includes an indicator of the student’s overall performance level 
in terms of one of four performance categories: unsatisfactory, partially proficient, 
proficient, and advanced. In addition, the report provides a scale score associated with 
the overall performance level, with scale score intervals defined for each of the four 
performance categories. Lastly, the report breaks down the overall performance level 
by content standard and subcontent area. The reports indicate whether the student’s 
performance was either below proficient or proficient or above in each of the content 
standards and subcontent areas. Unlike the ACT, no normative data is supplied with 
the report. Because of it’s design, the reports clearly indicate how well the performance 
matches up with the CMCS. 

School level diagnostic reports 

ACT ACT provides schools with a diagnostic report that indicates to school administrators 
the percentage of students falling into six score ranges (1 to 15, 16 to 19, 20 to 23, 
24 to 27, 28 to 32, and 33 to 36) in each of the four areas tested by the ACT. These 
percentages are juxtaposed to analogous values tabulated for the state in which the 
school resides, and to the nation overall. Again, because the ACT doesn’t map well 
onto the CMCS, inferences about how well the school is meeting the CMCS are not 
possible. 

It is important to note that the ACT report is for the graduating class of that year, 
not the students taking the ACT during the previously concluded academic year. If a 
student took the ACT during their junior year, their score is not summarized at the 
school level until after their senior year, when they became a member of the graduating 
class. Thus, the results presented in ACT diagnostic reports at the school level do not 
represent the results for students at a particular grade level. Though, overall, most 
students taking the ACT are seniors, s their results are combined with the juniors’ 
results from the previous academic year, the sophomores’ results from two academic 
years prior, etc. This differs from the way the CSAP exams aggregate student results, 
where results are reported for same grade level and the same academic year. 

CSAP CDE provides schools and districts with a diagnostic report that includes, for each 
subject and grade level, district percentages/counts of students at each of the four 
CMCS performance levels, as well as the percentage/count of students for which no 
score was reported. These percentages/counts are then disagregated by gender, race, 
handicapping condition, accommodations, program (i.e., IEP, 504 plan, and Title I), 
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time in district, time in school, migrant status, and language background. The stu- 
dents with no scores (both for the district and for each individual school in the district) 
are indicated in one of five categories: Does not read English or Spanish; Individual- 
ized standards; Parental refusal; One or more test sessions invalid; One or more test 
sessions incomplete. In addition, schools are provided with a list of individual student 
performance indicating each student’s score, performance level, and whether the stu- 
dent performed below proficient or at/above proficient on each of the CMOS content 
standards for the specific subject being tested. Like with the individualized CSAP 
reports, the design of the diagnostic report clearly links to CMOS performance. 



Conclusions 



The ACT does not match well with the CMOS, and the match is especially poor in science. 
This finding casts serious doubt on the foremost rationale for requiring all Colorado 11th 
graders to take the ACT, namely, that the ACT provides a good measure of how well students 
have mastered the CMCS. It should be noted that several other rationales are not impugned 
by the finding of a poor match between the ACT and the CMCS. These rationales are (1) 
that requiring all Colorado 11th graders to take the ACT might encourage more students 
to pursue a college education and (2) that taking the ACT in their junior years might help 
students identify deficiencies that they might remedy in their senior years. 

That the ACT is poorly matched with the CMCS has two further implications. First, 
it potentially injects incoherence into the CMCS-based accountability system as a whole. 
Because the CSAPs and the ACT measure different things, they are potentially at cross 
purposes. That is, teaching that concentrates on increasing performance on the CSAP exams 
could come at the cost of lowering performance on the ACT, and vice versa. Second, and 
related to the first problem, the poor match raises serious question about how ACT results 
might be combined with CSAP performance in the calculation of Student Accountability 
Reports for high schools. Because the reports are finely tuned to provide information related 
to performance on the CMCS, it is unclear how to amend the reports’ format to include 
ACT performance without reducing the reports’ focus on performance with respect to the 
CMCS. 



One important finding pertaining to an issue that was not included in the original aim 
of the study emerged through the process of data analysis. In particular, the 2002 ACT 
graduating class summary will indicate a decrease of approximately 2.5 points compared to 



2001. This is an outcome of the manner in which AC'! 



Inc. 



reports results coupled with 



all Colorado juniors being required to take the exam in 2001 (placing them in the 2002 
graduating class). This decrease should be interpreted with some caution since it is largely 



15 



20 



the result of a different population of students in Colorado taking the ACT. Comparisons to 
previous Colorado ACT graduating class results or to other states’ ACT results based upon 
self-selecting ACT examinees are tenuous. 
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